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ABSTRACT 

The  use  of  unmanned  air  vehicles  (UAVs)  in  military  operations  is  expanding  rapidly,  and  this  trend  will  likely 
continue  given  increases  in  funding  for  UAV  development  from  $3  billion  in  the  1990s  to  over  $12  billion  for  2004- 
2009.  High  UAV  mishap  numbers  have  generated  multiple  reviews  of  unmanned  operations  in  the  past  few  years, 
but  even  within  common  platforms,  different  analysts  attributed  these  mishaps  to  differing  causes.  Thirty  Air  Force 
Predator  Class  A  mishaps  (more  than  $1  million  damage)  occurred  from  the  introduction  of  this  system  into  the  Air 
Force  inventory  in  1995  through  the  end  of  FY  2006.  Reports  were  reviewed  to  identify  trends.  Substantial  changes 
over  time  were  observed  regarding  annual  mishap  rates,  annual  mishap  counts,  and  causal  factors.  Mishap  rates 
across  the  past  three  years  dropped  to  less  than  one  half  the  rate  across  earlier  years.  Mishap  counts ,  however, 
steadily  increased,  as  did  Predator  flying  hours.  Early  mishap  reports  typically  cited  mechanical  problems  and 
operator  station  design  issues.  Mechanical  problems  were  much  less  frequently  cited  in  the  last  three  years.  Rather, 
80%  of  recent  mishaps  cited  causal  human  error  factors.  Equipment  interface  problems  were  still  cited  as  causal  or 
major  contributing  factors  in  almost  half  of  recent  mishaps.  Recent  mishap  reports  often  cited  shortfalls  in  skill  and 
knowledge  (checklist  error,  task  misprioritization,  lack  of  training  for  task  attempted,  and  inadequate  system 
knowledge),  situation  awareness  (channelized  attention),  and  crew  coordination.  These  trends  come  in  a  period 
characterized  by  a  rapidly  growing  crew  force  and  highlight  the  need  to  revisit  both  individual  and  team  Predator 
training  objectives  and  consider  alternative  training  interventions  that  focus  on  the  practice  and  improvement  of 
these  key  operator  skill  areas.  Predator  team  coordination  and  situation  awareness  training  objectives  are  also 
addressed  for  the  command  and  control  personnel  with  whom  Predator  crews  interact. 
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INTRODUCTION 

The  Department  of  Defense  (DoD)  dictionary  (Joint 
Pub  1-02)  defines  unmanned  aerial  vehicles  (UAVs)  as 
“powered  aerial  vehicles  that  do  not  carry  a  human 
operator,  use  aerodynamic  forces  to  provide  vehicle 
lift,  can  fly  autonomously  or  be  piloted  remotely,  can 
be  expendable  or  recoverable,  and  carry  a  lethal  or 
non-lethal  load.”  While  early  attempts  to  apply  such 
technology  for  military  purposes  can  be  traced  back  to 
World  War  II  or  before  (Gambone,  2002),  UAVs 
clearly  entered  the  mainstream  of  combat  operations 
during  recent  conflicts  in  Kosovo,  Afghanistan,  and 
Iraq. 

In  these  recent  conflicts,  UAVs  were  typically  used  as 
Intelligence,  Surveillance,  and  Reconnaissance/Target 
Acquisition  assets,  providing  commanders  with 
imagery  intelligence,  electronic  intelligence,  and 
streaming  video.  Resulting  information  could  be  used 
to  direct  fighter  aircraft  to  their  targets,  monitor  enemy 
movements,  and  conduct  battle  damage  assessment. 
The  Predator  system  added  the  strike  mission  to  its 
repertoire,  and  plans  for  similar  capabilities  in  other 
DoD  UAVs  are  not  far  behind.  Additional  roles 
include  homeland  security  (e.g.,  border  patrol),  long- 
duration  law  enforcement  surveillance,  and  delivery  of 
critical  medical  supplies  needed  on  the  battlefield 
(Bone  and  Balkcom,  2003).  The  Quadrennial  Defense 
Review  (Office  of  the  Secretary  of  Defense,  2006) 
emphasized  that  approximately  45%  of  the  future  long- 
range  strike  force  will  be  unmanned  and  that  there 
would  be  a  clearly-defined  need  to  double  the  UAV 
coverage  capacity  by  accelerating  the  acquisition  of 
Predator  UAVs.  Predators  entered  the  Air  Force 
inventory  in  1995,  and  flying  hours  increased  rapidly, 
expanding  more  than  twenty-fold  in  the  decade  from 
1997  to  2006  (Air  Force  Safety  Center,  2007).  Despite 
this  rapid  growth,  only  about  one  third  of  requests  for 
Predator  surveillance  can  currently  be  met,  with  growth 
in  flying  hours  being  limited  by  the  ability  to  train 
enough  crews  to  meet  this  demand  in  Afghanistan  and 
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Iraq  (Brook,  2007).  Increases  in  planned  funding  are 
even  steeper  in  the  next  few  years  (Bone  and  Balkcom, 
2003).  UAVs  are  predicted  to  dominate  the  battlespace 
in  the  2 1 S1  century  and  will  include  intelligent, 
autonomous  systems  of  systems  that  are  expected  to 
conduct  a  myriad  of  missions  from  surveillance  and 
reconnaissance  to  suppression  of  enemy  air  defense 
and  precision  strike. 

While  there  is  considerable  demand  for  UAV  support, 
the  rapid  rise  in  UAV  employment  has  been 
accompanied  by  high  mishap  numbers.  Tvaryanas, 
Thompson,  and  Constable  (2005)  conducted  an  in- 
depth  review  of  UAV  mishaps  across  the  United  States 
military  services.  They  reported  that,  since  the 
inception  of  the  systems  through  the  end  of  FY  2003, 
334  mishaps  per  100,000  flying  hours  occurred  with 
the  Navy/Marine  Corps  Pioneer,  55  mishaps  per 
100,000  flying  hours  occurred  with  the  Army’s  Hunter 
system,  and  32  mishaps  per  100,000  flying  hours 
occurred  with  the  Air  Force’s  Predator  system.  For 
comparison  purposes,  overall  Air  Force  Class  A 
mishap  rates  ($1  million  damage  or  fatality)  are 
typically  in  the  low  single  digit  range  per  100,000 
flying  hours  (O'Toole,  Hughes,  &  Musselman,  2006). 
It  should  be  noted  that  most  manned  aircraft  are  mature 
systems,  while  most  UAV  programs  are  relatively  early 
in  their  life  cycles,  and  mishap  rates  tend  to  improve 
with  system  maturity.  Additionally,  many  UAV 
systems  did  not  undergo  a  classic  acquisition, 
development  and  fielding  program.  Many  were  fielded 
directly  from  development  into  operational  duty. 

Several  senior  reviewers  have  drawn  negative 
implications  for  UAV  affordability  and  mission 
availability  from  these  high  initial  mishap  rates.  An 
Office  of  the  Secretary  of  Defense  UAV  reliability 
study  (2003)  concluded  that  it  was  critical  to  improve 
UAV  reliability  because  affordability,  availability,  and 
acceptance  are  all  linked  to  reliability.  A  Defense 
Science  Board  Study  on  Unmanned  Aerial  vehicles  and 
Uninhabited  combat  Aerial  Vehicles  (2004)  concluded 
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that  UAV  programs  have  not  yet  expended  the 
resources  necessary  to  fix  the  root  causes  leading  to 
mishaps,  and  that  manned-aircraft-like  reliability  is 
achievable,  but  will  require  substantial  additional 
investment.  A  recent  challenge  from  the  Secretary  of 
Defense  to  reduce  the  numbers  of  mishaps  by  at  least 
50%  also  focused  attention  on  UAV  mishap 
frequencies.  In  the  Air  Force,  20%  of  Class  A  mishaps 
in  FY  2004-2006  involved  UAVs,  a  percentage  that 
remained  constant  across  all  three  years. 

A  consistent  picture  of  the  problem  to  be  solved  has 
not  yet  emerged  for  UAV  mishap  reduction.  Even  at 
basic  levels  such  as  the  relative  contributions  of 
equipment  failure  versus  human  error,  different 
analysts  reached  widely  differing  conclusions.  The 
Defense  Science  Board  (2004)  reported  that  17%  of 
UAV  mishaps  were  attributable  to  human  error,  while 
Tvaryanas  and  his  colleagues  (2005)  reported  that  68% 
of  mishaps  involved  causal  human  factors.  An  Office 
of  the  Secretary  of  Defense  Reliability  Study  (2003) 
reported  that  human  error  represented  16%  of  all 
sources  of  Predator  A  (MQ-1)  system  failures  and  2% 
of  Predator  B  (MQ-9)  mishaps,  while  Williams  (2004) 
reported  that  67%  of  Predator  mishaps  involve  human 
factors.  Some  researchers  looked  at  Class  A  (more  than 
$1  million  damage  or  a  fatality),  B  (more  than 
$200,000  damage,  and  C  (more  than  $20,000  damage) 
mishaps  (e.g.,  Tvaryanas,  2006),  some  considered 
Class  A  mishaps  only  (Williams,  2004),  and  others  did 
not  specify  the  scope  of  the  mishaps  analyzed. 

Experience  with  previous  efforts  to  reduce  mishaps  in 
manned  aircraft  dictates  that  successful  interventions  to 
improve  reliability  must  be  based  on  an  accurate 
understanding  of  the  root  causes  leading  to  failure. 
Several  researchers  recently  documented  differing  root 
cause  patterns  across  organizations  and  platforms. 
Helmreich,  Wilhelm,  Klinect,  and  Merritt  (2001) 
studied  threats  to  safety  and  the  nature  of  errors  in 
three  domestic  air  carriers  in  the  United  States,  and 
striking  differences  were  observed  among  these  airlines 
regarding  both  threats  to  safety  and  operator  errors 
despite  obvious  commonality  with  respect  to  mission 
and  environment.  Nullmeyer,  Stella,  Montijo,  and 
Harden  (2005)  reported  differing  mishap  root  causes 
across  Air  Force  manned  aircraft  types.  Williams 
(2004)  reported  major  deviations  in  root  causes  across 
UAVs,  and  Tvaryanas,  et  al.  reported  significant 
differences  among  root  causes  depending  on  the 
service  involved. 

Based  on  rapidly  increasing  UAV  operations,  the 
emphasis  from  senior  military  leaders  on  reducing 
UAV  mishaps,  and  the  lack  of  consensus  in  the 
literature  on  causal  factors,  we  felt  that  root  cause 


analyses  are  needed  to  assess  the  role  that  training 
interventions  could  play  to  reduce  mishaps  and 
increase  capability  for  a  given  platform.  Our  focus  in 
this  paper  is  on  root  causes  and  other  characteristics  of 
the  30  Air  Force  Predator  Class  A  mishaps  that 
occurred  from  FY  1997  through  the  end  of  FY  2006. 
This  initial  focus  was  chosen  in  part  because  Class  A 
mishap  reports  are  more  detailed  than  Class  B  or  Class 
C  mishap  reports,  and  in  part  because  Class  A  mishaps 
have  evolved  as  highly  visible  metrics  of  safety  and 
reliability.  Based  on  the  patterns  that  emerged  from 
these  analyses,  training  interventions  are  proposed  to 
address  the  areas  of  greatest  potential  gain. 

METHODS 

Class  A  flight  mishaps  (more  than  $1  million  damage 
or  a  fatality)  are  usually  investigated  by  two  groups,  a 
Safety  Investigation  Board  (SIB),  and  an  Accident 
Investigation  Board  (AIB).  The  primary  purposes  of 
safety  investigations  are  to  provide  timely  assessments 
of  possible  force-wide  implications  on  the  combat 
readiness  of  the  systems  involved  and  to  find  causes  in 
order  to  prevent  future  mishaps.  Safety  investigation 
reports  include  facts,  board  analyses,  findings,  causes, 
and  recommendations.  The  AIB  reviews  the  factual 
information  reported  in  the  safety  investigation  report, 
gathers  additional  information  as  deemed  necessary, 
and  provides  a  report  for  public  release  that  includes 
the  Board  President’s  opinion  about  what  caused  the 
mishap.  The  AIB  report  may  also  describe  factors  that 
are  believed  to  have  contributed  to  the  mishap.  Both 
SIB  and  AIB  data  were  reviewed  regarding  the  30  Air 
Force  Predator  Class  A  mishaps  that  occurred  from  the 
time  the  system  entered  the  Air  Force  inventory  in 
1995  through  the  end  of  FY  2006. 

Primary  Data  Sources 

The  United  States  Judge  Advocate  General’s  office 
maintains  an  online  repository  of  AIB  report 
summaries  for  Air  Force  Class  A  mishaps.  This  site 
(http://usaf.aib.law.af.mil)  is  publicly  accessible,  lists 
Class  A  mishaps  by  fiscal  year  across  the  Air  Force, 
and  provides  one  page  executive  summaries  of  AIB 
reports  as  they  are  released.  Contents  include  a 
description  of  the  mishap,  a  discussion  of  probable 
cause,  and  recommendations. 

Mishap  data  are  available  from  the  Air  Force  Safety 
Center  at  varying  levels  of  granularity.  The  analyses 
reported  here  considered  information  from  four  distinct 
Safety  Center  data  sources.  Moving  from  general  to 
specific,  the  first  was  statistical  data  from  the  Air  Force 
Safety  Center  web  site  (http://afsafety.af.mil).  These 
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data  include  hours  flown  and  numbers  of  Class  A 
mishaps  by  fiscal  year  and  by  aircraft  type. 

The  second  data  source  was  safety  investigation 
summaries.  These  documents,  a  few  pages  long, 
provided  a  brief  narrative  of  the  mishap,  and 
categorized  the  30  Predator  Class  A  mishaps  as  being 
logistics-,  maintenance-,  or  operations-related. 
Summaries  also  provide  descriptive  data  for  each 
mishap  such  as  phase  of  the  mission  and  time  of  day  in 
which  the  mishap  occurred.  Finally,  conclusions  and 
recommendations  were  listed. 

The  third  source  of  data  was  a  detailed  Human 
Factors  Database  that  is  maintained  by  Safety  Center 
Life  Sciences  analysts.  Through  2006,  analysts  used 
the  Mishap  Human  Factors  Taxonomy  to  structure 
findings  regarding  the  roles  played  by  operators, 
maintainers,  and  other  personnel.  This  taxonomy 
divides  human  factors  into  two  major  branches— 
environmental  or  individual  factors.  These  two 
branches  are  further  subdivided  as  shown  in  Figure  1 . 


Figure  1 :  Mishap  Human  Factors  Taxonomy 


Each  area  is,  in  turn,  further  divided  into  sub-areas.  For 
example,  "Operations"  includes  mission  preparation, 
cockpit/crew  resource  management  training,  procedural 
guidance/publications,  and  mission  demands.  Finally, 
several  elements  comprise  each  sub-area,  resulting  in 
over  360  detailed  elements  that  are  available  to  catalog 
mishap  human  factors.  Each  factor  cited  is  assigned  a 
weight  using  the  following  scale:  (4)-causal;  (3)-major 
factor,  (2)-minor  factor,  (l)-minimal  factor,  or  (0)- 
present  but  not  a  factor. 

The  fourth  and  most  detailed  data  source  in  the  current 
analysis  was  discussions  of  human  factors  from  the  full 
mishap  investigation  reports.  SIB  Findings  are 


formally  documented  as  one  section  of  the  full  mishap 
report.  These  findings  were  reviewed  for  descriptions 
of  human  factors  causing  or  contributing  to  the  mishap. 
In  addition  to  the  board  findings,  a  separate  Life 
Sciences  Report  is  prepared  by  the  Life  Sciences 
Branch  of  the  Air  Force  Safety  Center.  The  Life 
Sciences  Report  provides  a  chronological  mishap 
narrative  and  a  discussion  of  every  human  factors 
database  element  cited.  Interrelationships  among  the 
human  factors  may  be  addressed. 

Analytic  Approach 

AIB  summary  reports  and  Safety  Center  summary 
statistics  were  initially  analyzed  to  generate  descriptive 
trend  data  regarding  mishap  frequencies,  mishap  rates, 
and  the  general  nature  of  mishaps  (equipment  failure, 
maintainer  error,  or  operator  error)  over  time.  Mishap 
frequency  and  flying  hour  data  were  obtained  from  the 
Air  Force  Safety  Center  web  site 
(http://afsafety.af.mil).  Data  regarding  probable  cause 
were  obtained  from  the  Air  Force  Judge  Advocate 
General  Corps  web  site  (http://usaf.aib.law.af.mil).  We 
completed  our  overall  trend  analyses  by  assigning  each 
Predator  Class  A  mishap  to  the  phase  of  flight  in  which 
it  occurred  based  on  data  from  Safety  Center  mishap 
summaries. 

The  specific  human  factors  elements  cited  in  Predator 
Class  A  mishaps  were  obtained  from  the  Air  Force 
Safety  Center  human  factors  database.  This  database 
lists  all  human  factors  cited  in  the  Life  Sciences  Report 
section  of  each  full  mishap  report  and  provides  a 
weighted  score:  4=causal,  3=major  factor,  2=minor 
factor,  l=minimal  factor,  and  0=present  but  not  a 
factor.  From  this  database,  we  determined  both 
frequencies  of  occurrence  and  levels  of  contribution 
associated  with  each  detailed  human  factors  element. 
Sums  of  weighted  factors  were  calculated  across 
mishaps  for  each  Human  Factors  Taxonomy  element 
cited.  These  weighted  sums  were  used  to  rank-order  the 
individual  elements.  Most  of  the  detailed  elements  that 
were  cited  fit  into  seven  general  categories:  operator 
interface  issues,  skill/knowledge,  documentation, 
planning/  preparation,  and  organizational  or 
management  issues.  As  an  example,  crew  coordination, 
crew  composition,  intracockpit  communication  and 
subordinate  style  were  combined  to  form  the  teamwork 
category.  These  groupings  allowed  identification  of 
trends  among  similar  individual  factors. 

The  human  factors  database  also  guided  subsequent 
reviews  of  the  Life  Science  Reports  since  every  factor 
cited  in  the  database  was  discussed  in  the  full  reports. 
Qualitative  analyses  of  these  discussions  were 
accomplished  to  gain  a  better  understanding  of  the 
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underlying  behaviors  that  led  to  each  element  being 
cited.  There  is  considerable  risk  of  misinterpretation  if 
database  quantitative  analyses  are  accomplished 
without  reviewing  the  descriptive  content  of  the 
associated  Life  Sciences  Division  reports. 

RESULTS 

Predator  flying  hours  per  year  increased  substantially 
over  the  time  period  covered  in  this  analysis  (Figure  2) 
as  reported  on  the  Air  Force  Safety  Center  Web  site 
(http://afsafety.af.mil).  Annual  flying  hours  increased 
from  less  than  3000  in  1997  to  almost  60,000  in  2006. 
Projections  call  for  continuing  increases  in  UAV 
operations.  These  changing  utilization  levels  are 
important  to  consider  when  interpreting  trends  in 
mishap  frequencies  over  time.  Predator  mishap 
frequencies  also  increased  over  fiscal  years  as  shown  in 
Figure  3  (http://usaf.aib.law.af.mil,  2007),  with  fiscal 
years  accounting  for  over  65%  of  the  variability 
observed  in  mishap  frequencies  (correlation  =  .81,  p  < 
.001).  The  primary  cause  of  the  mishaps  appears  to  be 
shifting  over  time.  Causes  in  AIB  reports  were 
normally  stated  in  terms  of  equipment  failure,  operator 
error,  or  a  combination  of  the  two  and  are  depicted  by 
fiscal  year  in  Figure  3.  Mishaps  in  the  first  few  years 
were  often  attributed  to  equipment  failures,  while 
mishaps  in  the  past  three  years  were  predominantly 
attributed  to  operator  error.  A  similar  pattern  was  seen 
in  Safety  Center  mishap  summaries.  Fifteen  mishaps 
occurred  between  1997  and  2003,  and  15  additional 
mishaps  occurred  between  2004  and  2006,  where  a 
statistically  significant  change  was  seen  from  the  first 
15  mishaps  being  attributed  primarily  to  problems  not 
related  to  operators  (logistics  or  maintenance)  to 


mishaps  being  attributed  primarily  to  operator  error  in 
the  most  recent  15  mishaps  (chi  square  =  5.40,  df  =  1,  p 
<  .02).  In  Safety  center  analyses,  nine  of  the  first  15 
mishaps  were  attributed  to  equipment  factors,  and  even 
four  of  the  six  operator-error  mishaps  cited  causal 
equipment  interface  problems.  In  total,  thirteen  of  the 
fifteen  mishaps  from  1997  through  2003  cited  causal 
equipment  factors.  In  the  second  half,  one  mishap  was 
attributed  to  equipment  failure  and  the  remaining  14 
were  attributed  to  operator  (12  mishaps)  or  maintainer 
(2  mishaps)  error.  Further,  only  three  of  the  1 1  mishaps 
attributed  to  operations  cited  equipment  as  a  causal 
factor.  Functional  design  continued  to  be  cited  as  a 
contributing  factor,  however,  in  many  of  these  recent 
mishaps.  Finally,  UAV  mishaps  represented 
approximately  20%  of  all  Air  Force  Class  A  mishaps  in 
each  of  the  past  three  years  (http://usaf.aib.law.af.mil). 

Predator  mishap  rates  per  100,000  flying  hours,  on  the 
other  hand,  decreased  substantially  over  the  same  time 
period  (Figure  4).  Despite  comments  from  some 
reviewers  of  unusually  high  mishap  rates  in  UAV 
systems.  Predator  mishap  rates  are  following  a  pattern 
that  is  very  similar  to  the  rates  seen  early  in  the  history 
of  the  F-16  weapon  system,  which  also  encountered 
both  mechanical  and  human  error  problems  early  in  its 
life  cycle.  F-16  mishaps  are  now  very  close  to  overall 
Air  Force  mishap  rates.  In  Figure  4,  both  the  predator 
and  the  historic  F-16  mishap  rates  depicted  start  with 
the  first  year  in  which  more  than  5000  hours  were 
flown  annually  in  that  platform.  For  F-16  mishaps,  the 
data  points  represent  the  time  period  between  1977  and 
1984.  For  comparison  purposes,  the  overall  Air  Force 
mishap  rate  has  been  slightly  less  than  two  mishaps  per 
100,000  flying  hours  for  the  past  decade  or  more. 


Figure  2:  Predator  Hours  Flown 


Fiscal  Year 
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Figure  3:  Predator  Class  A  Mishap  Frequencies  and  Causes 

(http://usaf.aib.law.af.mil) 


£ 


Fiscal  Year 


Safety  Center  mishap  summaries  were  reviewed  to 
identify  the  mission  phases  in  which  the  mishaps 
occurred.  Results  are  shown  in  Figure  5.  The 
sequence  of  events  leading  to  the  Class  A  mishaps 
started  during  the  enroute  phase  of  the  mission  more 
than  two  thirds  of  the  time.  It  should  be  noted  that 
Predator  missions  may  last  many  hours,  and  with 
respect  to  elapsed  time,  this  phase  accounts  for  a  huge 
proportion  of  the  hours  flown.  Of  some  interest,  both 
equipment  failures  and  human  error  were  well 
represented  in  this  phase.  Most  other  mishaps  occurred 
in  the  final  approach  and  landing  phases  of  the  mission, 
and  these  typically  involved  operator  error. 


We  next  analyzed  the  specific  human  factors  elements 
that  were  cited  in  Safety  Center  Class  A  mishap 
reports.  Approximately  360  specific  elements 
comprised  the  human  factors  taxonomy  summarized  in 
Figure  1.  This  taxonomy  was  used  by  the  Safety  Center 
during  the  time  period  of  interest  (FY1997-2006). 
Analysts  used  72  of  these  360  potential  elements  to 
capture  the  human  factors  associated  with  the  30  Class 
A  Predator  mishaps  that  occurred  through  the  end  of 
2006.  Each  human  factor  cited  was  accompanied  by  a 
weighting  that  reflected  the  level  of  contribution  of  that 
factor  to  the  outcome  (4  =  causal,  3  =  major  factor,  2  = 
minor  factor,  and  1=  minimal  factor). 
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Figure  4:  Predator  Class  A  Mishap  Rates 
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For  each  individual  taxonomy  element,  numbers  of 
mishaps  where  that  element  was  cited  were  combined 
with  factor  weightings  to  develop  a  prioritized  list  of 
underlying  factors.  The  top  ten  elements  are  listed  in 
Table  1.  Major,  minor,  and  minimal  factors  were 
combined  to  form  the  contributing  factor  counts.  Of 
the  30  total  mishaps,  the  Safety  Center  attributed  17  to 
operations.  Each  of  the  17  reports  of  operations-related 
mishaps  cited  multiple  human  factors,  and  several  cited 
multiple  causal  human  factors. 

Many  of  the  individual  taxonomy  elements  cited  in 
Predator  mishaps  appeared  to  be  highly  related.  For 
example,  two  commonly  cited  individual  factors  listed 
in  Table  1  were  channelized  attention  and  inattention. 
Both  of  these  seem  to  reflect  insufficient  awareness  of 
the  operating  environment.  Further  review  of 
individual  human  factors  revealed  that  most  factors 
cited  could  be  accommodated  using  seven  higher-order 
categories:  (1)  equipment-related  issues  such  as 


functional  design,  operator  interface,  and  software 
logic;  (2)  decision  making  and  risk  assessment;  (3) 
operator  skills  and  knowledge;  (4)  situation  awareness; 
(5)  teamwork;  (6)  documentation  such  as  technical 
orders  and  written  procedures;  and  (7)  mission 
preparation  activities. 

Given  the  magnitude  of  changes  in  mishap  rates  over 
time,  it  seemed  reasonable  to  see  if  some  early  factors 
had  been  resolved.  To  do  this,  a  first  half/second  half 
comparison  was  accomplished  on  the  human  factors 
cited  in  Predator  Class  A  mishaps.  Figure  6  shows  the 
frequencies  with  which  elements  from  the  eight 
categories  were  cited  as  causal  or  major  contributing 
factors  in  the  first  15  Predator  mishaps,  which  occurred 
from  FY  1997  through  the  end  of  FY  2003.  Equipment 
interface  issues  were  the  most  commonly  cited  human 
factors  in  these  early  mishaps,  followed  by  situation 
awareness  factors,  decision  making  or  risk  assessment, 
and  lack  of  adequate  written  procedures. 


Figure  5:  Predator  Class  A  Mishaps  by  Mission  Phase 


Individual  Factor 

Area 

Causal 

Contributing 

Written  Procedures 

Documentation 

5 

6 

Channelized  Attention 

Situation  Awareness 

4 

5 

Functional  System  Design 

Equipment 

4 

4 

Checklist  Error 

Skill/Knowledge 

3 

3 

Crew  Coordination 

Teamwork 

2 

4 

Course  of  Action  Selected 

Decision  Making 

4 

1 

No  Training  for  Task 

Skill/Knowledge 

1 

4 

Inattention 

Situation  Awareness 

2 

3 

Task  Misprioritization 

Skill/Knowledge 

1 

4 

Automation 

Equipment 

2 

2 

Table  1:  Top  Ten  Detailed  Mishap  Factors 
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Figure  6:  Early  (1997-2003)  Mishap  Factor  Categories 

(9  Operations-Related  Mishaps) 


□  Contributing 

□  Causal 


Causal  and  major  contributing  factors  cited  in  more 
recent  15  Predator  Class  A  mishaps  (FY  2004  -  2006) 
are  summarized  in  Figure  7.  The  major  categories 
remained  the  same  as  those  in  the  earlier  mishaps,  but 
there  appeared  to  be  some  shifts  of  emphasis  among 
the  categories.  Equipment  and  decision  making  factors 
were  still  frequently  cited,  but  usually  described  as 
contributing  to  causal  operator  error.  There  was  a 
substantial  increase  in  instances  where  skill/knowledge 
and  teamwork  issues  were  judged  to  be  causal 
compared  to  the  patterns  seen  in  earlier  mishaps. 

Specific  elements  contributing  to  increased  skill  and 
knowledge  factors  were  crews  not  properly  following 
checklists,  no  training  for  task  attempted,  inadvertent 
inputs  to  the  Predator  through  the  operator  station,  and 
limited  total  experience.  Lack  of  written  procedures, 
functional  design  issues  associated  with  controls  and 
displays,  and  lapses  in  getting  software  change  notices 
to  crews  frequently  contributed  to  these  mishaps. 

Teamwork  issues  involved  several  failures  of  the  pilot 
to  react  to  inputs  from  the  sensor  operator  or  a  lack  of 
one  crewmember  to  back  up  another.  Several  involved 
confusion  between  instructor  pilots  and  student  pilots 
regarding  who  had  control  of  the  aircraft.  In  several 
cases,  there  is  no  written  procedure  regarding  roles, 
especially  when  troubleshooting.  Until  recently, 
simulation  did  not  support  crew-level  training. 

DISCUSSION 

Predator  mishap  trends  reflected  systematic  and 
substantial  changes  over  time.  The  overall  direction  of 


Predator  mishap  trends  depends  on  the  measure  used. 
Mishap  frequencies  steadily  increased  over  time  as 
have  Predator  hours  flown,  while  mishap  rates 
decreased  substantially  (from  23  Class  A  mishaps  per 
100,000  flying  hours  from  fiscal  years  1997-2003  to 
less  than  11  in  fiscal  years  2004-2006).  Predator 
mishap  rates  remain  high  relative  to  more  mature  Air 
Force  weapon  systems,  but  they  are  similar  to  the  rates 
seen  in  the  early  years  of  F-16  operations  and  are 
dropping  quickly.  We  believe  that  mishap  rates  are  the 
most  relevant  global  trend  data.  While  rates  are  still 
high,  they  are  moving  toward  the  rates  seen  in  other, 
more  mature  Air  Force  platforms.  With  respect  to  why 
Predators  crash,  a  steady  shift  was  observed  from 
equipment  failures  to  human  error.  In  the  past  three 
years,  we  found  a  substantial  increase  in  mishaps  that 
cite  a  lack  of  the  operator  skills  and  knowledge  needed 
to  successfully  cope  with  the  events  encountered  in  real 
world  operations. 

The  threat  and  error  management  model  (Helmreich,  et 
al,  2001)  is  widely  used  by  air  carriers  to  enhance 
safety.  It  also  provides  a  reasonable  structure  for 
improving  UAV  mishap  rates  in  military  operations 
and  ultimately  for  increasing  combat  capability.  A  key 
part  of  this  approach  is  to  use  evidence  to  structure 
interventions  that  are  tailored  to  alleviate  the  specific 
problems  that  actually  plague  a  particular  community. 
Training  is  one  of  several  tools  that  can  be  used  to  meet 
safety  and  capability  objectives,  but  other  changes  such 
as  equipment  modifications  and  adapted  procedures 
may  also  be  integral  parts  of  an  effective  overall  error 
mitigation  strategy.  The  bottom  line  is  that  the  better 
we  understand  threats  to  safety,  the  more  successful 
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Figure  7:  Later  (2004-2006)  Mishap  Factor  Categories 
(12  Operations-Related  Mishaps) 


□  Contributing 

□  Causal 


✓  * 
/  o*° 


<5> 

sPV 


I  ■  a  ■ 


1 


jy  £t- 


we  are  likely  be  in  developing  effective  strategies  to 
mitigate  human  error.  Mishap  trends  to  date  appear  to 
reflect  two  separate  generations  of  factors — equipment 
failures  in  the  early  years  followed  by  a  series  of 
human  error  mishaps  spanning  the  last  three  years.  A 
third  generation  of  Predator  training  opportunities  will 
likely  accompany  the  move  by  all  services  toward 
network-centric  warfare  (NCW). 

Attention  was  drawn  to  the  first  generation  of  threats  to 
safety  by  early  reviewers  who  voiced  concerns  about 
reliability  of  Predator  power  plants  and  other  aircraft 
systems.  With  the  vast  majority  of  early  mishaps 
attributed  to  such  equipment  failures,  this  was 
reasonable  and  accurate.  Trends  over  the  past  decade 
suggest  that  these  early  risk  reduction  efforts  are 
paying  off — mishap  frequencies  and  rates  due  to 
equipment  failure  have  both  steadily  improved. 
Operations-related  mishaps  are  driving  current  mishap 
counts  and  rates,  but  four  Predator  mishaps  were 
attributed  to  maintainer  errors  including  two  in  the  last 
three  years.  Threat  and  error  management  is  not  just  an 
operator  issue.  There  is  growing  recognition  that 
human  factors  training  for  maintainers  has  merit,  and  is 
often  referred  to  as  maintenance  resource  management. 

Mishap  trend  data  suggest  that  the  Predator  community 
has  entered  a  second  generation  of  mishaps,  now  fueled 
by  operator  error.  Can  these  human  error  causes  be 
addressed  by  training?  The  implications  for  training 
run  the  gamut  from  the  way  they  are  trained  to  what 
type  of  training  they  receive. 


Personal  observations  over  the  past  5  years  indicate  a 
decrease  in  the  experience  levels  of  Predator  aircrews. 
During  the  early  establishment  of  the  Predator  weapons 
system,  when  crew  resource  management  (CRM) 
training  was  conducted,  the  aircrews  were  generally 
very  experienced  in  their  previous  weapons  systems. 
Pilots  had  at  least  two  if  not  three  or  more  tours  in  a 
particular  weapons  system.  Sensor  operators  were 
generally  of  mid-level  rank  and  had  at  least  two 
previous  assignments,  many  of  them  aviation  related, 
prior  to  coming  to  the  Predator.  Starting  in  2004, 
Predator  CRM  training  went  from  approximately  60 
per  year  to  over  100  and  doubled  again  in  FY  2006  to 
216  students  per  year.  From  personal  interviews  and 
observations,  the  experience  level  of  an  average  pilot  is 
now  one  operational  tour,  and  at  least  50  percent  of 
new  sensors  came  directly  from  basic  training. 

Predators  are  operated  by  two  kinds  of  teams.  A 
launch  and  recovery  team  accomplishes  take  offs, 
approaches,  and  landings,  while  a  mission  crew 
handles  the  enroute  surveillance,  reconnaissance,  target 
acquisition,  and  potentially,  attack  functions.  An 
exception  to  this  pattern  is  formal  training  sorties, 
when  all  mission  phases  are  accomplished  by  one 
instructor  and  student  crew.  The  mishap  trends  suggest 
that  there  is  ample  opportunity  to  improve  performance 
in  both  teams. 

Predator  Class  A  landing  mishaps,  generally  involved 
bounced  landings  with  damage  to  the  landing  gear  and 
sensor  ball  with  resulting  damage  being  in  excess  of 
reporting  limits  for  a  Class  A  mishap.  Control  delays, 
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narrow  field  of  view,  and  the  software  interface 
between  the  ground  control  station  and  the  air  vehicle 
are  all  significant  threats  to  safety  that  are  faced  by 
launch  and  recovery  crews.  Of  some  interest,  all  Class 
A  landing  mishaps  have  occurred  in  the  past  three 
years,  suggesting  that  a  review  of  changes  in  selection, 
training,  or  operating  procedures  may  pay  dividends. 
This  period  also  coincides  with  the  rapid  expansion  in 
the  Predator  weapons  system,  when  the  number  of 
Predator  crews  more  than  doubled.  Perceptual  errors 
and  pilot-induced  oscillation  are  frequently  discussed. 
In  addition,  lapses  of  judgment  often  appear  in  the 
form  of  continuing  a  marginal  attempt  to  land  when  a 
go-around  was  the  better  choice. 

The  enroute  portion  of  flight  is  the  most  likely  mission 
phase  for  Class  A  mishaps  to  occur.  Both  equipment 
failure  and  operator  error  are  most  likely  to  result  in  a 
Class  A  mishap  enroute.  The  large  amounts  of  time 
spent  in  this  phase  undoubtedly  contribute  to  the  high 
proportions  of  mishaps.  This  is  also  where  the  “meat” 
of  the  mission  occurs  and  activity  levels  and  mission 
types  can  vary  widely.  This  phase  can  range  from 
routine,  such  as  orbiting  for  hours  watching  a  target 
with  no  appreciable  activity,  to  a  high  intensity  period 
launching  Hellfire  missiles  in  an  urban  close  air 
support  scenario.  Aircrews  also  spend  a  large  amount 
of  time  transiting  between  various  targets  and 
responding  to  short  notice  requests  by  ground  assets  for 
immediate  surveillance.  Supervisory  control  issues 
such  as  the  use  of  autopilot  hold  modes  and  changing 
preprogrammed  waypoints  appear  frequently  in 
discussions  of  enroute  Class  A  mishaps. 

Current  Predator  syllabus  training  leads  to  a  basic 
qualification  to  operate  the  weapons  system  for  the 
“enroute”  phase  of  flight.  Takeoff  and  landing 
qualifications  are  accomplished  at  a  later  date.  A 
qualified  crew  in  theater  normally  accomplishes  the 
takeoff  and  hands  the  Predator  off  to  the  mission  crew 
for  the  actual  operational  mission  and  then  receives  the 
aircraft  back  for  the  landing.  Additionally,  other 
qualifications  such  as  Air  Strike  Control  and  Close  Air 
Support  are  added  with  “top-off’  training.  These  are 
complex  applications  of  the  weapons  system  and 
performance  is  probably  highly  influenced  by  a  pilot’s 
previous  operational  experience.  Some  pilots  may  have 
little  or  no  experience  in  the  type  of  mission  flown  by 
the  Predator  (e.g.  tanker  or  cargo  pilots),  which  is 
likely  to  have  a  major  impact  on  training  requirements. 

Many  operator  errors  leading  to  Predator  mishaps  are 
associated  with  AFI  11-290  CRM  skills,  including  task 
management,  situation  awareness,  decision  making, 
and  crew  coordination  (teamwork).  Research  has 
shown  that  these  skills  can  be  improved  with  targeted 


training  (Salas,  et.  al,  2006).  Currently  there  is  an  on¬ 
going  research  effort  to  further  refine  the  specific 
behaviors  involved  in  these  mishaps  and  more 
importantly  develop  associated  training  solutions  to 
improve  performance.  There  are  many  potential 
training  programs  and  exercises  that  can  teach  aircrews 
about  distraction,  inattention,  channelized  attention, 
task  prioritization,  checklist  management,  decision 
making/risk  management,  and  crew  coordination. 

The  mishap  factor  category  that  has  shown  the  largest 
increases  over  time  is  skill/knowledge.  While  this  has 
been  traditionally  associated  with  task  management 
under  Air  Force  CRM  definitions,  there  are  other 
implications  for  training  in  this  area  as  well.  The 
current  Predator  training  syllabus  most  likely  needs 
improvement  to  bolster  the  minimum  required 
knowledge  to  operate  the  Predator  weapons  system. 
Whether  this  takes  the  form  of  additional  classes,  more 
in-depth  classroom  study,  self  study,  additional 
computer  based  training,  or  additional  testing  is  best 
determined  by  further  analysis. 

The  third  generation  is  to  move  Predator  training 
beyond  a  two-person  domain  and  provide  a  more 
realistic  representation  of  the  real  world.  The  Air 
Force  web  site  (www.af.mil),  defines  the  Predator 
system  as  follows:  “The  MQ-1  Predator  is  a  system, 
not  just  an  aircraft.  A  fully  operational  system  consists 
of  four  aircraft  (with  sensors),  a  ground  control  station, 
a  Predator  Primary  Satellite  Link,  and  approximately 
55  personnel  for  deployed  24-hour  operations.” 
Beyond  this  large  team,  Predator  operations  are  part  of 
a  much  larger  command  and  control  network. 

The  emergence  of  skill,  knowledge,  and  teamwork 
factors  leading  to  Predator  mishaps  suggests  these 
same  skill,  knowledge  and  teamwork  issues  are  present 
more  broadly  in  Predator  operations.  Especially  given 
the  frequency  with  which  mishap  reports  cite  lack  of 
training  for  task  attempted,  a  training  architecture  to 
address  these  same  issues  within  the  larger  command 
and  control  environment  seems  to  make  sense.  A  key 
element  of  this  architecture  is  a  training  environment 
that  includes  sensor  feeds  and  interactive  simulations  to 
increase  knowledge,  build  teamwork,  and  hone  critical 
decision-making  skills.  The  adverse  trends  in  operator 
error  and  crew  coordination  could  be  used  to  shape 
decision-making  skills  through  NCW-command/ 
control  scenarios.  Effective  command  and  control  is 
inherently  an  iterative  decision-making  process  as 
feedback  from  the  battlespace  is  incorporated  into 
plans  and  corrective  actions.  Warfare  has  always  been 
a  challenging  domain  characterized  by  the  importance 
of  the  endeavor,  risk  to  life,  sheer  magnitude  of  effort, 
and  management  of  uncertainty. 
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While  approaches  to  command  and  control  have  been 
honed  over  time  to  include  improved  training,  there 
still  lacks  a  training  environment  to  exercise  the  entire 
network.  Information  Age-driven  changes  present  us 
with  a  host  of  new  command  and  control  challenges. 
These  changes  challenge  our  most  basic  assumptions 
about  command  and  control  based  on  a  doctrine 
developed  from  a  different  time  to  solve  a  different 
problem.  One  of  the  most  enduring  lessons  derived 
from  the  history  of  warfare  is  the  degree  to  which  fog 
and  friction  permeate  the  battlespace.  The  fog  of  battle 
is  about  the  uncertainty  associated  with  what  is  going 
on,  while  the  friction  of  war  is  about  the  difficulty  in 
translating  the  commander’s  intent  into  actions.  Much 
of  the  fog  of  war,  or  what  is  referred  to  today  as  the 
lack  of  battlespace  awareness,  have  resulted  in  our 
inability  to  tap  into  our  collective  knowledge,  or  the 
ability  to  assemble  existing  information,  reconcile 
differences,  and  construct  a  common  picture.  There 
needs  to  be  equal  emphasis  placed  upon  developing  a 
current  awareness  of  both  friendly  and  enemy 
dispositions  and  capabilities,  as  well  as  increased 
emphasis  on  neutrals. 

In  summary,  high  UAV  mishap  rates  have  generated 
much  high  level  attention.  Our  review  of  Predator 
Class  A  mishaps  revealed  that  most  recent  mishaps 
involve  a  limited  range  of  threats  to  safety  and  a 
manageable  set  of  causal  and  contributing  human 
factors.  These  results  suggest  that  training  can  be 
focused  on  solving  a  finite  set  of  human  performance 
problems.  Human  factors  trends  suggest  that  training 
is  becoming  an  increasingly  crucial  tool  for  reducing 
mishaps.  The  solution  will  likely  involve  revisions  to 
formal  school  syllabus  content  as  well  as  to  sharpening 
the  focus  of  targeted  human  factors  training.  While 
operator  error  is  commonly  cited  in  Predator  mishaps, 
value  is  also  apparent  in  maintenance  resource 
management  training.  Finally,  the  next  logical  step  is 
to  readdress  teamwork  training  for  the  larger  command 
and  control  team  in  which  Predator  crews  operate. 

RFEFERENCES 

Air  Force  Safety  Center  (2007).  Flight  Statistics. 

Obtained  from  http://afsafetv.af.mi1/stats/f  stats. asp. 
Bone,  E.  &  Bolkcom,  C.  (2003).  Unmanned  Aerial 
Vehicles:  Background  and  issues  for  Congress.  25 
April,  2003.  www.fas.org/irp/crs/RL31872.pdf. 
Brooks,  T.V.  (2007).  More  Predators  Sought,  but 
Service  Short  on  Crew.  Air  Force  Times,  30  March 
2007,  http://www.airforcetimes.com/news/20Q7/03/ 
gns  predator  short  070329 /. 

Defense  Science  Board  (2004).  Defense  Science  Board 
Study  on  Unmanned  Aerial  Vehicles  and 


Uninhabited  Combat  Aerial  Vehicles.  Washington 
DC:  Office  of  the  Undersecretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics. 

Department  of  Defense  (2001).  Department  of  defense 
dictionary  of  military  and  associated  terms.  Joint 
Publication  1-02. 

Gambone,  G.  (2002).  From  U.S.  civil  war  to 
Afghanistan:  a  shore  history  of  UAVs.  Defense  LINK 
News,  April,  2002. 

Helmreich,  R.L.,  Wilhelm,  J.A.,  Klinect,  J.R.,  & 
Merritt,  A.C.  (2001).  Culture,  error,  and  crew 
resource  management.  In  E.  Salas,  C.A.  Bowers,  & 
E. Edens  (Eds.),  Applying  resource  management  in 
organizations:  A  guide  for  professonals.  Hillsdale, 
NJ:  Erlbaum. 

Nullmeyer,  R.,  Stella,  D.,  Montijo,  G.A.,  and  Harden, 
S.W.  (2005).  Human  factors  in  air  force  flight 
mishaps:  Implications  for  change.  Proceedings  of  the 
Interservice/Industry  Training,  Simulation i,  and 
Education  Conference,  Orlando  FL. 

Office  of  the  Secretary  of  Defense  (2003).  Unmanned 
Aerial  Vehicle  Reliability  Study.  February,  2003. 
www.acq.osd.mil/uas/docs/reliabilitystudy.pdf. 

Office  of  the  Secretary  of  Defense  (2006).  The 
Quadrenial  Defense  Review  Report,  February  6, 
2006.  http://www.globalsecurity.org/militarv/librarv/ 
policy/dod/qdr-2006-report.pdf. 

O’Toole,  K.J.,  Hughes,  T.,  and  Musselman,  B.T. 
(2006)  USAF  Aviation  Safety:  FY  2005  in  Review. 
http://afsafety.af.mil/SEF/Downloads/FY05  Life  Sc 

iences  Safety  Review(AsMA06).ppt 

Rumsfeld,  D.  (2003)  Reducing  Preventable  Accidents. 
Memorandum  for  Secretaries  of  the  Military 
Departments.  The  Secretary  of  Defense,  Washington 
DC. 

Salas,  E.,  Wilson,  K.A.,  Burke,  C.S.,  and  Wightman, 
D.C.  (2006).  Does  crew  resource  management 
training  Work?  An  undate,  an  extension,  and  some 
critical  needs.  Human  factor,  vol.  48  number  3,  page 
381. 

Tvaryanas,  A.,  Thompson,  B.T.,  and  Constable,  S.H. 
(2005).  U.S.  military  unmanned  aerial  vehicle 
mishaps:  assessment  of  the  role  of  human  factors 
using  HFACS.  Brooks  AFB,  TX:  311  Human 
Systems  Wing. 

Tvaryanas  and  Thompson  (2006).  Unmanned  Aircraft 
System  (UAS)  Operator  Error  Mishaps:  An 
Evidence-based  Prioritization  of  Human  Factors 
Issues.  RTO-MP-HFM- 135.  (NATO  report). 

Williams,  K.W.  (2004).  A  summary  of  unmanned 
aircraft  accident/incident  data:  human  factors 
implications.  Oklahoma  City,  OK:  Civil  Aerospace 
Medical  Institute.  Federal  Aviation  Administration. 


2007  Paper  No.  7133  Page  12  of  12 


